The pitfalls of questionnaire surveys
Practical Action
WHO
Who is the
beneficiary?
WHAT
What do you need to
know?
(develop hypothesis)
WHY
Why do you
need to know?
HOW
How will you get the information? (solve hypothesis)
Figure 1: Process to follow before selecting your methods for gaining information
Common pitfalls
Although, questionnaire surveys are still a common method of data collection, but there are a
number of important considerations and potential pitfalls you need to take into account.
Some of these pitfalls include:
Preparation: Initial research into the community and their values does not often
happen. What may be important to one person may not be a priority to another and
without investigation often key observations are not made before the questionnaire is
applied. Often there is a lack of engagement with the community from the out set.
Sample groups: What is an appropriate sample size? When large numbers are used
then long term monitoring is difficult to achieve. Field tests are crucial and if not
trialed it is unknown if the questionnaire is relevant and suitable for the specified
sample group. Localized differences between communities can mean that a survey
developed for one community may not be an appropriate proxy in another.
Not enough thought goes in to who will give you the required outputs - are they
young, old, men, women, teenagers, children under 5, mothers, fathers, unemployed,
small business owners? There is no point asking the wrong people for information. For
example asking the male head of the household “how long it takes to get water” when
it is the women and children who collect it.
Length: Often questionnaires are too lengthy. This can lead to confusion, people can
get bored, the questions can be invasive and inaccurate information can result.
Questions: Often the questions are not clear and concise or relevant to the topic;
there are conceptual gaps when using terminology and no thought of how it will be
understood by those receiving it; differences in interpretation; varying levels of
consistency between each survey. Below are some examples of inappropriate
questions:
1. Asking those living in urban or rural areas who don’t receive a set income and
rely on many different ways of making a living “What is your yearly income?”
2. Asking informal traders in slum areas “Are you a registered business?”
3. Asking slum dwellers “what is their integrated waste system within their home
and how much waste do they generate in a day: 1-2kg, 2-5kg?” This is a question
that would be difficult for anyone to answer.
There are often confusions with translation, for example “what time does it take you
to cook dinner” can be easily misinterpreted as “What time did you start cooking”.
This will affect your end result therefore careful consideration needs to be taken
when designing questions.
Qualitative vs quantitative: Through using only quantitative methods one type of
information is gathered without taking into consideration the quality of life of the
beneficiaries or their social relationships, aspirations and individual values. Often it
is not an empowering process for those taking part as it is not a participatory process
but technical in nature.
2